{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch：张量(Tensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy是一个很棒的框架，但是它不支持GPU以加速运算。现代深度神经网络，GPU常常提供[50倍以上的加速]((https://github.com/jcjohnson/cnn-benchmarks))，所以NumPy不能满足当代深度学习的需求。 \n",
    "\n",
    "我们先介绍PyTorch最基础的概念：**张量（Tensor）**。逻辑上，PyTorch的tensor和NumPy array是一样的：tensor是一个n维数组，PyTorch提供了很多函数操作这些tensor。任何希望使用NumPy执行的计算也可以使用PyTorch的tensor来完成；可以认为它们是科学计算的通用工具。\n",
    "\n",
    "和NumPy不同的是，PyTorch可以利用GPU加速。要在GPU上运行PyTorch张量，在构造张量使用`device`参数把tensor建立在GPU上。\n",
    "\n",
    "这里我们利用PyTorch的tensor在随机数据上训练一个两层的网络。和前面NumPy的例子类似，我们使用PyTorch的tensor，手动在网络中实现前向传播和反向传播： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 32397586.0\n",
      "1 29653808.0\n",
      "2 31517672.0\n",
      "3 32702292.0\n",
      "4 29156884.0\n",
      "5 21014596.0\n",
      "6 12371127.0\n",
      "7 6481777.0\n",
      "8 3437418.25\n",
      "9 2026104.125\n",
      "10 1369511.5\n",
      "11 1031340.0\n",
      "12 830772.5\n",
      "13 694495.5\n",
      "14 592425.5625\n",
      "15 511290.5\n",
      "16 444623.875\n",
      "17 388864.78125\n",
      "18 341748.375\n",
      "19 301557.875\n",
      "20 267141.28125\n",
      "21 237480.625\n",
      "22 211721.28125\n",
      "23 189273.0\n",
      "24 169670.421875\n",
      "25 152539.5\n",
      "26 137481.21875\n",
      "27 124183.4765625\n",
      "28 112407.890625\n",
      "29 101948.546875\n",
      "30 92621.1015625\n",
      "31 84301.3671875\n",
      "32 76848.7421875\n",
      "33 70166.609375\n",
      "34 64159.265625\n",
      "35 58752.2890625\n",
      "36 53871.515625\n",
      "37 49460.359375\n",
      "38 45468.28515625\n",
      "39 41847.3671875\n",
      "40 38560.2265625\n",
      "41 35574.5\n",
      "42 32857.10546875\n",
      "43 30378.65234375\n",
      "44 28117.48046875\n",
      "45 26050.265625\n",
      "46 24159.33203125\n",
      "47 22426.75390625\n",
      "48 20836.775390625\n",
      "49 19376.80859375\n",
      "50 18036.22265625\n",
      "51 16802.80078125\n",
      "52 15666.5712890625\n",
      "53 14618.611328125\n",
      "54 13651.189453125\n",
      "55 12757.51953125\n",
      "56 11931.091796875\n",
      "57 11166.2353515625\n",
      "58 10457.806640625\n",
      "59 9800.892578125\n",
      "60 9191.345703125\n",
      "61 8625.2431640625\n",
      "62 8099.2421875\n",
      "63 7609.904296875\n",
      "64 7154.30712890625\n",
      "65 6729.8349609375\n",
      "66 6335.0927734375\n",
      "67 5967.416015625\n",
      "68 5624.2939453125\n",
      "69 5303.5478515625\n",
      "70 5003.6923828125\n",
      "71 4723.2783203125\n",
      "72 4460.83154296875\n",
      "73 4214.8837890625\n",
      "74 3984.311767578125\n",
      "75 3768.076171875\n",
      "76 3565.220703125\n",
      "77 3374.729248046875\n",
      "78 3195.743408203125\n",
      "79 3027.56787109375\n",
      "80 2869.463623046875\n",
      "81 2720.654296875\n",
      "82 2580.54052734375\n",
      "83 2448.58642578125\n",
      "84 2324.1845703125\n",
      "85 2206.866455078125\n",
      "86 2096.213134765625\n",
      "87 1991.7703857421875\n",
      "88 1893.142822265625\n",
      "89 1799.97607421875\n",
      "90 1711.9854736328125\n",
      "91 1628.7353515625\n",
      "92 1549.9886474609375\n",
      "93 1475.492919921875\n",
      "94 1404.97705078125\n",
      "95 1338.1744384765625\n",
      "96 1274.890625\n",
      "97 1214.9462890625\n",
      "98 1158.0865478515625\n",
      "99 1104.175048828125\n",
      "100 1053.04541015625\n",
      "101 1004.4970092773438\n",
      "102 958.4166259765625\n",
      "103 914.675048828125\n",
      "104 873.1033935546875\n",
      "105 833.5908813476562\n",
      "106 796.0438232421875\n",
      "107 760.3388671875\n",
      "108 726.3853759765625\n",
      "109 694.0946044921875\n",
      "110 663.3565673828125\n",
      "111 634.0916748046875\n",
      "112 606.2379760742188\n",
      "113 579.708740234375\n",
      "114 554.4349365234375\n",
      "115 530.3472900390625\n",
      "116 507.39739990234375\n",
      "117 485.5131530761719\n",
      "118 464.6553955078125\n",
      "119 444.8115234375\n",
      "120 425.87945556640625\n",
      "121 407.8146667480469\n",
      "122 390.578857421875\n",
      "123 374.1240234375\n",
      "124 358.41656494140625\n",
      "125 343.41510009765625\n",
      "126 329.0862121582031\n",
      "127 315.39837646484375\n",
      "128 302.3253173828125\n",
      "129 289.8228759765625\n",
      "130 277.8758544921875\n",
      "131 266.45501708984375\n",
      "132 255.53192138671875\n",
      "133 245.0852813720703\n",
      "134 235.0945281982422\n",
      "135 225.5386505126953\n",
      "136 216.39132690429688\n",
      "137 207.6426544189453\n",
      "138 199.2645263671875\n",
      "139 191.24319458007812\n",
      "140 183.56752014160156\n",
      "141 176.21559143066406\n",
      "142 169.1734619140625\n",
      "143 162.430908203125\n",
      "144 155.97067260742188\n",
      "145 149.78070068359375\n",
      "146 143.8531036376953\n",
      "147 138.16989135742188\n",
      "148 132.72291564941406\n",
      "149 127.50228881835938\n",
      "150 122.4962387084961\n",
      "151 117.69648742675781\n",
      "152 113.09529113769531\n",
      "153 108.6822280883789\n",
      "154 104.44880676269531\n",
      "155 100.390869140625\n",
      "156 96.49755859375\n",
      "157 92.76048278808594\n",
      "158 89.17646026611328\n",
      "159 85.73511505126953\n",
      "160 82.43333435058594\n",
      "161 79.26356506347656\n",
      "162 76.22187805175781\n",
      "163 73.30120849609375\n",
      "164 70.49832153320312\n",
      "165 67.8067626953125\n",
      "166 65.2218017578125\n",
      "167 62.739776611328125\n",
      "168 60.35589599609375\n",
      "169 58.06590270996094\n",
      "170 55.866981506347656\n",
      "171 53.75346374511719\n",
      "172 51.72381591796875\n",
      "173 49.7740478515625\n",
      "174 47.90001678466797\n",
      "175 46.099151611328125\n",
      "176 44.36842727661133\n",
      "177 42.70504379272461\n",
      "178 41.10645294189453\n",
      "179 39.569313049316406\n",
      "180 38.0921516418457\n",
      "181 36.67244338989258\n",
      "182 35.30720520019531\n",
      "183 33.99496841430664\n",
      "184 32.73215866088867\n",
      "185 31.518840789794922\n",
      "186 30.351478576660156\n",
      "187 29.228731155395508\n",
      "188 28.148761749267578\n",
      "189 27.1102352142334\n",
      "190 26.110733032226562\n",
      "191 25.149866104125977\n",
      "192 24.22553253173828\n",
      "193 23.335954666137695\n",
      "194 22.480079650878906\n",
      "195 21.65622329711914\n",
      "196 20.86427116394043\n",
      "197 20.101438522338867\n",
      "198 19.36749267578125\n",
      "199 18.6611328125\n",
      "200 17.981666564941406\n",
      "201 17.327533721923828\n",
      "202 16.6976318359375\n",
      "203 16.09160614013672\n",
      "204 15.508203506469727\n",
      "205 14.946392059326172\n",
      "206 14.405319213867188\n",
      "207 13.88435173034668\n",
      "208 13.38334846496582\n",
      "209 12.90029239654541\n",
      "210 12.435243606567383\n",
      "211 11.987588882446289\n",
      "212 11.556377410888672\n",
      "213 11.140989303588867\n",
      "214 10.741242408752441\n",
      "215 10.356196403503418\n",
      "216 9.985044479370117\n",
      "217 9.62756061553955\n",
      "218 9.283443450927734\n",
      "219 8.95161247253418\n",
      "220 8.632147789001465\n",
      "221 8.32425308227539\n",
      "222 8.027596473693848\n",
      "223 7.74207067489624\n",
      "224 7.466590881347656\n",
      "225 7.201577663421631\n",
      "226 6.945904731750488\n",
      "227 6.699846267700195\n",
      "228 6.462244033813477\n",
      "229 6.233642101287842\n",
      "230 6.013014316558838\n",
      "231 5.8004889488220215\n",
      "232 5.595724105834961\n",
      "233 5.398106098175049\n",
      "234 5.208057880401611\n",
      "235 5.024584770202637\n",
      "236 4.847821235656738\n",
      "237 4.677356719970703\n",
      "238 4.512961387634277\n",
      "239 4.354593276977539\n",
      "240 4.201796531677246\n",
      "241 4.054564476013184\n",
      "242 3.9124152660369873\n",
      "243 3.7755191326141357\n",
      "244 3.6437957286834717\n",
      "245 3.516598701477051\n",
      "246 3.3939054012298584\n",
      "247 3.2756576538085938\n",
      "248 3.1615967750549316\n",
      "249 3.051727294921875\n",
      "250 2.9454429149627686\n",
      "251 2.8433165550231934\n",
      "252 2.7444913387298584\n",
      "253 2.649336338043213\n",
      "254 2.5574846267700195\n",
      "255 2.4689009189605713\n",
      "256 2.3834540843963623\n",
      "257 2.30104660987854\n",
      "258 2.2215287685394287\n",
      "259 2.1448192596435547\n",
      "260 2.0707833766937256\n",
      "261 1.999433159828186\n",
      "262 1.930531620979309\n",
      "263 1.8641365766525269\n",
      "264 1.7999260425567627\n",
      "265 1.7380388975143433\n",
      "266 1.6784169673919678\n",
      "267 1.6206767559051514\n",
      "268 1.5650670528411865\n",
      "269 1.511476993560791\n",
      "270 1.4597465991973877\n",
      "271 1.4097900390625\n",
      "272 1.3614823818206787\n",
      "273 1.3149127960205078\n",
      "274 1.2700271606445312\n",
      "275 1.2267100811004639\n",
      "276 1.1848374605178833\n",
      "277 1.1444698572158813\n",
      "278 1.105466365814209\n",
      "279 1.067838430404663\n",
      "280 1.0315113067626953\n",
      "281 0.9963598251342773\n",
      "282 0.9626108407974243\n",
      "283 0.9299044609069824\n",
      "284 0.8982887268066406\n",
      "285 0.8678793907165527\n",
      "286 0.8384674787521362\n",
      "287 0.8100439310073853\n",
      "288 0.7826417684555054\n",
      "289 0.7561572194099426\n",
      "290 0.7306160926818848\n",
      "291 0.7059524059295654\n",
      "292 0.6821449995040894\n",
      "293 0.6590884923934937\n",
      "294 0.6368918418884277\n",
      "295 0.6154152154922485\n",
      "296 0.5947210192680359\n",
      "297 0.5746666193008423\n",
      "298 0.5554118156433105\n",
      "299 0.5367658138275146\n",
      "300 0.5187095403671265\n",
      "301 0.5012819766998291\n",
      "302 0.48446130752563477\n",
      "303 0.4682067632675171\n",
      "304 0.45248138904571533\n",
      "305 0.43733394145965576\n",
      "306 0.42270106077194214\n",
      "307 0.40856367349624634\n",
      "308 0.3949173092842102\n",
      "309 0.3817325234413147\n",
      "310 0.36898136138916016\n",
      "311 0.3566476106643677\n",
      "312 0.3448025584220886\n",
      "313 0.333279550075531\n",
      "314 0.32217997312545776\n",
      "315 0.31147849559783936\n",
      "316 0.30107423663139343\n",
      "317 0.29105526208877563\n",
      "318 0.2813987731933594\n",
      "319 0.2720847725868225\n",
      "320 0.2630632519721985\n",
      "321 0.2543545067310333\n",
      "322 0.24592186510562897\n",
      "323 0.2377665936946869\n",
      "324 0.22991251945495605\n",
      "325 0.22228585183620453\n",
      "326 0.21490931510925293\n",
      "327 0.20781612396240234\n",
      "328 0.20096543431282043\n",
      "329 0.1943194717168808\n",
      "330 0.18794003129005432\n",
      "331 0.18174177408218384\n",
      "332 0.17575621604919434\n",
      "333 0.16996923089027405\n",
      "334 0.1643589437007904\n",
      "335 0.15896709263324738\n",
      "336 0.15375858545303345\n",
      "337 0.14868833124637604\n",
      "338 0.14381222426891327\n",
      "339 0.13910606503486633\n",
      "340 0.13453835248947144\n",
      "341 0.13010713458061218\n",
      "342 0.1258593499660492\n",
      "343 0.12172406911849976\n",
      "344 0.11773461103439331\n",
      "345 0.11389746516942978\n",
      "346 0.11018949747085571\n",
      "347 0.10659147799015045\n",
      "348 0.1031065583229065\n",
      "349 0.09974153339862823\n",
      "350 0.09648153930902481\n",
      "351 0.09333153814077377\n",
      "352 0.09028969705104828\n",
      "353 0.08736664056777954\n",
      "354 0.08451689779758453\n",
      "355 0.0817766934633255\n",
      "356 0.07910621911287308\n",
      "357 0.07654337584972382\n",
      "358 0.0740606039762497\n",
      "359 0.0716794803738594\n",
      "360 0.06935499608516693\n",
      "361 0.06710180640220642\n",
      "362 0.06492079794406891\n",
      "363 0.06282824277877808\n",
      "364 0.060792192816734314\n",
      "365 0.058823540806770325\n",
      "366 0.056920669972896576\n",
      "367 0.05509084835648537\n",
      "368 0.053311243653297424\n",
      "369 0.05159778892993927\n",
      "370 0.04994425177574158\n",
      "371 0.048335686326026917\n",
      "372 0.04677193611860275\n",
      "373 0.04528346657752991\n",
      "374 0.04381948336958885\n",
      "375 0.04241698980331421\n",
      "376 0.04107467830181122\n",
      "377 0.03975784033536911\n",
      "378 0.03847987949848175\n",
      "379 0.03724490478634834\n",
      "380 0.03605663776397705\n",
      "381 0.03489261120557785\n",
      "382 0.03378477692604065\n",
      "383 0.03271189332008362\n",
      "384 0.03167524188756943\n",
      "385 0.030669676139950752\n",
      "386 0.029691677540540695\n",
      "387 0.028751127421855927\n",
      "388 0.02783174440264702\n",
      "389 0.026950640603899956\n",
      "390 0.026094920933246613\n",
      "391 0.02526796981692314\n",
      "392 0.024473119527101517\n",
      "393 0.02369825914502144\n",
      "394 0.02295231819152832\n",
      "395 0.022242795675992966\n",
      "396 0.02153945341706276\n",
      "397 0.02086133137345314\n",
      "398 0.02020978182554245\n",
      "399 0.019581736996769905\n",
      "400 0.018969284370541573\n",
      "401 0.01837668940424919\n",
      "402 0.017801767215132713\n",
      "403 0.017242692410945892\n",
      "404 0.016707751899957657\n",
      "405 0.0161900632083416\n",
      "406 0.015690064057707787\n",
      "407 0.015199274756014347\n",
      "408 0.014730056747794151\n",
      "409 0.014279086142778397\n",
      "410 0.013840025290846825\n",
      "411 0.013414962217211723\n",
      "412 0.01300005055963993\n",
      "413 0.01260110829025507\n",
      "414 0.012217523530125618\n",
      "415 0.011838346719741821\n",
      "416 0.01147967204451561\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "417 0.011122584342956543\n",
      "418 0.010785242542624474\n",
      "419 0.010456754826009274\n",
      "420 0.010140564292669296\n",
      "421 0.0098289605230093\n",
      "422 0.009533515200018883\n",
      "423 0.009244212880730629\n",
      "424 0.008960932493209839\n",
      "425 0.008697972632944584\n",
      "426 0.008437827229499817\n",
      "427 0.008179335854947567\n",
      "428 0.00793493539094925\n",
      "429 0.0076987105421721935\n",
      "430 0.007467738352715969\n",
      "431 0.007244306616485119\n",
      "432 0.0070322174578905106\n",
      "433 0.006825670599937439\n",
      "434 0.006624545902013779\n",
      "435 0.006430486217141151\n",
      "436 0.0062418594025075436\n",
      "437 0.00605994276702404\n",
      "438 0.0058792466297745705\n",
      "439 0.005711874924600124\n",
      "440 0.005545766558498144\n",
      "441 0.0053859129548072815\n",
      "442 0.0052306982688605785\n",
      "443 0.00507597578689456\n",
      "444 0.00493097398430109\n",
      "445 0.004789197817444801\n",
      "446 0.004654687829315662\n",
      "447 0.0045255254954099655\n",
      "448 0.004393784794956446\n",
      "449 0.004269877448678017\n",
      "450 0.004152118694037199\n",
      "451 0.004035870544612408\n",
      "452 0.003924779128283262\n",
      "453 0.0038115603383630514\n",
      "454 0.003706799354404211\n",
      "455 0.0036034109070897102\n",
      "456 0.003505885833874345\n",
      "457 0.0034097759053111076\n",
      "458 0.0033171125687658787\n",
      "459 0.0032301023602485657\n",
      "460 0.003142215311527252\n",
      "461 0.003059015143662691\n",
      "462 0.002976488322019577\n",
      "463 0.0028983764350414276\n",
      "464 0.0028228582814335823\n",
      "465 0.002749705919995904\n",
      "466 0.002678253687918186\n",
      "467 0.002608206355944276\n",
      "468 0.002539908280596137\n",
      "469 0.002474888227880001\n",
      "470 0.0024095058906823397\n",
      "471 0.0023496721405535936\n",
      "472 0.0022876225411891937\n",
      "473 0.0022305548191070557\n",
      "474 0.002173817716538906\n",
      "475 0.002120976336300373\n",
      "476 0.002067095134407282\n",
      "477 0.0020155953243374825\n",
      "478 0.001963778166100383\n",
      "479 0.0019166870042681694\n",
      "480 0.0018687359988689423\n",
      "481 0.0018243305385112762\n",
      "482 0.0017811936559155583\n",
      "483 0.0017367161344736814\n",
      "484 0.0016962094232439995\n",
      "485 0.0016547166742384434\n",
      "486 0.001614880282431841\n",
      "487 0.0015773135237395763\n",
      "488 0.0015398586401715875\n",
      "489 0.001504470594227314\n",
      "490 0.0014690652024000883\n",
      "491 0.001436532475054264\n",
      "492 0.001401829533278942\n",
      "493 0.0013696056557819247\n",
      "494 0.001338304951786995\n",
      "495 0.001309270621277392\n",
      "496 0.0012771730544045568\n",
      "497 0.0012488742358982563\n",
      "498 0.001220754231326282\n",
      "499 0.0011930970940738916\n"
     ]
    }
   ],
   "source": [
    "# 可运行代码见本文件夹中的 two_layer_net_tensor.py\n",
    "import torch\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') \n",
    "\n",
    "# N是批大小； D_in 是输入维度；\n",
    "# H 是隐藏层维度； D_out 是输出维度\n",
    "N, D_in, H, D_out = 64, 1000, 100, 10\n",
    "\n",
    "# 产生随机输入和输出数据\n",
    "x = torch.randn(N, D_in, device=device)\n",
    "y = torch.randn(N, D_out, device=device)\n",
    "\n",
    "# 随机初始化权重\n",
    "w1 = torch.randn(D_in, H, device=device)\n",
    "w2 = torch.randn(H, D_out, device=device)\n",
    "\n",
    "learning_rate = 1e-6\n",
    "for t in range(500):\n",
    "    # 前向传播：计算预测值y\n",
    "    h = x.mm(w1)\n",
    "    h_relu = h.clamp(min=0)\n",
    "    y_pred = h_relu.mm(w2)\n",
    "\n",
    "    # 计算并输出loss；loss是存储在PyTorch的tensor中的标量，维度是()（零维标量）；\n",
    "    # 我们使用loss.item()得到tensor中的纯python数值。\n",
    "    loss = (y_pred - y).pow(2).sum()\n",
    "    print(t, loss.item())\n",
    "\n",
    "    # 反向传播，计算w1、w2对loss的梯度\n",
    "    grad_y_pred = 2.0 * (y_pred - y)\n",
    "    grad_w2 = h_relu.t().mm(grad_y_pred)\n",
    "    grad_h_relu = grad_y_pred.mm(w2.t())\n",
    "    grad_h = grad_h_relu.clone()\n",
    "    grad_h[h < 0] = 0\n",
    "    grad_w1 = x.t().mm(grad_h)\n",
    "\n",
    "    # 使用梯度下降更新权重\n",
    "    w1 -= learning_rate * grad_w1\n",
    "    w2 -= learning_rate * grad_w2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Spyder)",
   "language": "python3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "227.797px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
